Segatron: Segment-Aware Transformer for Language Modeling and Understanding

Transformers are powerful for sequence modeling. Nearly all state-of-the-art language models and pre-trained based on the Transformer architecture. However, it distinguishes sequential tokens only with token position index. We hypothesize that better contextual representations can be generated from richer positional information. To verify this, we propose a segment-aware (Segatron), by replacing original encoding combined of paragraph, sentence, token. first introduce mechanism to Transformer-XL, which is popular Transformer-based model memory extension relative encoding. find our method further improve Transformer-XL base large model, achieving 17.1 perplexity WikiText-103 dataset. investigate pre-training masked modeling task Segatron. Experimental results show BERT Segatron (SegaBERT) outperform vanilla various NLP tasks, outperforms RoBERTa zero-shot sentence representation learning. Our code available GitHub.